Friday, November 25, 2011

Data Overload: How to Create a Better System?

Deed scans waiting to be processed
I have always stored electronic genealogy files on my computer.  Up until this year my system worked pretty well. I had a main directory for my mother's surname and another for my father's.  Under that I had subdirectories for further surnames.  I would just sort of bung everything together in one surname directory.

My main organizational method included a file naming pattern such as this: SurnameGivenName-Year-documenttype.  That would translate as EdwardsJohn-1792-baptism.jpg.  In this way I could see all my ancestors alphabetically, then chronologically by name and also quickly identify the specific document.  I thought I was pretty clever using that system.

Now I don't feel so clever.  The thing is I'm experiencing data overload.  In the early days when I was simply downloading images from Ancestry.com that system worked pretty well.  Now I am regularly taking photographs, scanning papers copies on my non-portable flatbed scanner, scanning from books and records with my Magic Wand and next up will be even more scanning, including microfilm, with the Flip Pal.

As far as I can see I have three challenges:

1) Processing all the electronic files and renaming them
2) Organizing them in a more specific manner (considering that the volume has increased 10 fold)
3) Deciding which files to attach to a genealogy software program

(Another challenge which I won't even try to address here is how to do all of the above for house histories which don't fit neatly into the family focused programs.)

My number one goal is to be able to put my hands on a specific file at a moment's notice. If I can achieve that, I will feel like I have a good system in place.  I would also like to be able to process all of my electronic files efficiently.  Right now I am a bit overwhelmed at the thought of renaming the hundreds that I can bring in during just one research trip.

Is anyone else dealing with this enormous jump in the intake of electronic files?  How have you coped with increase in volume? How are you renaming your files? Is anyone using a database to manage their files? Or is there perhaps a software program that helps keep the chaos under control?

I would love to hear your suggestions!

22 comments:

  1. Marian, I have a similar problem, so I completely understand the overwhelming feeling of too much data. Since my computer crashed I am looking at how I store, file and maintain my digital files. I look forward to reading other responses and suggestions. Thanks for the post!

    ReplyDelete
  2. I have a similar system of surname folders, and within each one I have folders for BDMs, Correspondence (from family members), Directories, Family Histories (from family members), Land, Newspapers, People (photos), Places (photos), Probate, and so on. I can find what I need fairly quickly, even if the individual file doesn't have a helpful name.

    Good luck!

    ReplyDelete
  3. Marian, WOW when you figure it out let us all know! ~laughs~ With two deaths this past summer, I have inherited so much paper and photos. My previous system does not seem to be working well.

    I watched a Webinar at Legacy on "Heritage Collector" and I will be trying that software out. Hopefully it will help. The system I have been using is pretty much like yours, Surname and sub-folders of the different documents, with names, subject and dates. With so many subscriptions and saving so many documents, I am overwhelmed and feel that very shortly my current system will not work.

    I look forward to hearing others comments, thank you for posting!

    ReplyDelete
  4. Although I do have sort of a naming system, in sorting my image files I depend on my folder system to keep all of these files straight. Folders are arranged by family name, then by generation, then into the individual family units within that generation, and then into something resembling Carol's system above - correspondence, BDM, land, photos, etc.

    ReplyDelete
  5. MRIN Filing - files and folders can grow and grow and grow and it never wears out. With the MRIN as the identifying prefix, the rest of the file name is not critical. An approximate identification will do. And you'll know exactly where to go to find a specific document. Combine that with keyword, (source, copyright, etc) annotation and you're good to go. For that, something like GeoSetter, Photo Mechanic, XnView. A lot can be done in batch-mode.

    ReplyDelete
  6. I have been working with my system for a long time and it works just fine for me. I have created a web page which attempts to explain this system.- I can find most any of my documents, images, etc. instantly -
    http://freepages.genealogy.rootsweb.ancestry.com/~robbhaas/Misc/MyOrganizationSystem.htm

    ReplyDelete
  7. Marian, I started by using binders with a tabbed section for each Surname. I could never figure out what to do when a woman married and acquired a new surname, and then what if she married again? I soon out grew the binders (and space for them) and became frustrated with the whole system. In the last 6 months I've been filing almost exclusively electronically with Evernote. What I like about Evernote is that I can create a folder for each surname then each document that I save in that folder can be tagged making it much easier to retrieve. For example documents for my Great Grandmother, Mary Evingham Smith will be saved in the Evingham folder but they are tagged with Evingham, Smith, Illinois, and Nebraska and Ohio. She was born in Ohio but lived and was on the census in Nebraska and Ohio. So if I want to see all of my ancestors that were enumerated in Nebraska I just pull up that tag. Evernote does allow for subfolders if you wish to set them up as well. The best part is I have access to everything no matter where I am or what device I am on...desktop, laptop or phone. So far I love it, the key is to remember to tag your documents.

    ReplyDelete
  8. Hi, a colleague in Ireland has a similar system involving having the surname in each filename. He then uses the search box in Windows 7 to quickly find info relating to names. He showed me the system recently and it is very quick. He is very suspicious of web-based systems and I can see why he is happy with his Windows 7 and MyBrothersKeeper combination.
    I have been using OneNote but your (and his) folder based system seems more robust.
    Thanks for such a relevant discussion.

    ReplyDelete
  9. When I first got started I followed the format recommended by Lisa Louise Cook of GenealogyGems. It has a lot of subfolders within the surnames and can be a bit time consuming to find things. Plus I suspect I have some multiple copies. Anyway, for the file renaming/ processing I started using a free Bulk Renaming Utility program http://www.bulkrenameutility.co.uk/Main_Intro.php It offers a lot of control and is speedy (& free!).

    Best of luck!

    ReplyDelete
  10. I'm just entering the paper and electronic hunter/gatherer phase of life as I verify those on my already extended pedigree. No physical documentation was passed on so I'd like to know about the perfect filing system before inundation occurs. It should work for me, too. Half of my pedigree is "Smith" and "Bishop" where given names repeat in each generation. The other half is Scandinavian where a surname system didn't even exist before 1825 or so. Here, too, few given names were used. Female names were simply the male name with a feminine suffix and many people picked up a new moniker at will. Hopefully, filing systems exist or are developed soon that are effective, efficient, and foolproof for everyone! Please continue informing us.

    Thanks to all who contribute! Della

    ReplyDelete
  11. Basically I have eight folders, great grandparents. Inside each folder is a folder for census images, vital records, photos - whatever you need. Sometimes I include a special folder if there is a branch I have a lot on. I heard a podcast a year or so ago by Lisa Cooke. Her system is pretty much the same as mine and she offers some good tips.

    I also have same surnames [unrelated] in different lines and then same surnames [unrelated] who married so I have line 1, line 2 and then the merged line. DNA now indicates that about 1720 they were related, not sure what happens when I find that ancestor. :-)

    ReplyDelete
  12. I use evernote (love it!) and there is also a similar android app called catch that is pretty good.

    ReplyDelete
  13. I also use the Surname folder system. I have created folders for each of my 16 g-g-grandparents. My twist is that I do not create subfolders for record types. I create subfolders for any person who has more than one document or image. I do not group the subfolders by family group. I name the subfolders as "surname, first birthyear-deathyear." I set the view up as a list so that it looks like a name index when I open each folder. My individual files names start with "Surname, First document type/name" so that they show up in the list alpabetically if there is no subfolder for that person yet.

    I started renaming the files in March and am still working on it. The reason it is taking so long (besides life interfering) is that I am adding citations to each image or document in the comments section of the file properties.

    ReplyDelete
  14. O Lord, Sierra. Have you considered the backup of those 'file properties'? If you're working on Windows XP that could be major trouble. Windows7 I'm not sure.

    ReplyDelete
  15. My system is very much like Greta's and Carol’s. I start off first with four main folders called “Father’s Father”, “Father’s Mother”, “Mother’s Father” and “Mother’s Mother”. Within each of these folders, I break it down by surname, such as Farrell (under Father’s Father), then within that, a sub-folder for each family (such as William Forbes Farrell Family), then within each family I have sub-folders for BMD's, passenger lists, correspondence, etc). Even if I don’t give the files an overly descriptive name, I can find what I am looking for quite easily.

    ReplyDelete
  16. Jen - thanks so much for the link to that bulk renaming utility program.

    ReplyDelete
  17. I find it hard to work with all these folders within folders within folders, especially when I get to the upper branches of my family tree and can't quite remember which sub-sub-sub folder that particular family surname would be in.

    So I settled on a folder for each grandparent surname and within that have a subfolder for every subsequent surname on that branch of the tree. Then I name files like you with SurnameGivenname - date - document type. So even if I want something from my 5th great grandparents, it is at most three clicks away.

    ReplyDelete
  18. Just wondering. How does the folder system work when you have people from one of your grandparents' lines married into people from another of your grandparents' lines? Do you file them in both? Or how do you remember which one you chose?

    ReplyDelete
  19. Early on, because I started off inputting a published Hoisington genealogy and I wanted to link pictures and source material, I knew I would need some kind of a system to keep everyone's information straight. My genealogy program gives each individual a fixed number when I enter the data.

    The system I worked out is a combination of given names and a number, i.e. Jane_10341. I would use this in combination with the type of information, i.e.Jane_10341_obit.jpg or Jane_10341_mc.jpg (marriage certificate or Jane_10341_house.jpg (for a picture of a house or a deed) or Jane_10341_news.txt (a newspaper article). Each of these can be modified if more than one obit or newspaper article or even marriage exists by adding _1 or _2, etc.

    This system works as is for most genealogy program (it does have to have a fixed numbering system). I have 18,000+ individuals in my website's database and have not run into any problems yet. I imagine it would work for a paper system if you wanted to keep a separate index by Surname.

    ReplyDelete
  20. I have a system that works for me - each document is named on the back as soon as I get it - a birth cert for Janet Smith from 1867 would be B1867 Janet Smith. I enter this as the document name in my genealogy program and add each piece of data from the certificate. If you then need to look at the doc again, you can check the "source" and know where to look in your folders. Jo

    ReplyDelete
  21. Lots of good information here.... Have you decided on a "system" yet, Marian? If so, I (and probably others) would be interested in knowing what you came up with.

    ReplyDelete
  22. I started out with 4 folders (Lisa's maternal, Lisa's paternal and my husband's maternal & paternal). Under each of those folders are surname folders. Inside each of the surname folders are generation folders such as "01James Gorrell&SarahMilholland". The earliest folder begins with 00. (I know, I don't know what I'll do if I find the parents of James Gorrell--I've looked for 20 years and haven't found it yet). Inside these folders are the files. They begin with the date of the record such as "1821DeathSarahGorrell-RossCoOH.tif". I like to use the date first so everything is in chronological order and I can see what I'm missing, especially with census records. I might have additional folders for non-direct line children inside the parent's folder.

    My paper file system is set up the same way. I've been working on updating both systems the past few months. It's a big job!

    I, too, will be interested in what you decide to do, Marian.

    ReplyDelete