The next crucial bit of information is the state where he's likely to have joined up. You see, for the most part, the soldiers did not serve directly in the Federal Army, nor in the Confederate Army, but in a unit of the army organized from volunteers in their home state. The basic unit formed in a state was called a regiment - they had names like "Tenth Indiana Infantry" or "2nd Georgia Cavalry" or "1st Maine Hvy Arty," which is military speak for "Heavy Artillery." Note that from border states like Tennessee, both the Union and Confederacy had units called "3rd Tennessee Infantry," so it will narrow your search if you know which side he was on.
You'll need to find out your soldier's regiment if you want to get his records from the National Archives, or read about where he fought. The following tips may help you do that.
The reels of microfilm have the soldiers' names in alphabetical order - what you'll see on the film are individual index cards with the soldier's name and the name of a unit he served in.
The names on these lists are not soundexed - if you're looking for "Arrowood", you'll have to check "Airwood" and "Arawod" and so forth - you know the drill.
I'll start with the confederates - they're easy, because they're all on one combined list, "Consolidated Index to compiled service records of Confederate soldiers," National Archives Film Series M-253. In the LDS locality catalog, look under "United States - Military Records - Civil War, 1861-1865 - Indexes"
Well, they're not really all on that list. That's an index to papers in the possession of the federal government. Many confederate soldiers will only be found in the papers of their native states. Sorry, I don't know much about how to find them there.
There's a separate microfilm series for each state. Look for "Index to Compiled Service Records of Union Soldiers Who Served from Indiana." In the LDS locality catalog, that would be under "Indiana - Military Records." The index card will contain the soldier's name and a unit he served in. Check the southern states, too - every state had units of Union volunteers.
The big bonanza for Union soldiers is the index of pension applications - invalid pensions, widow's pensions, orphan's pensions, mother's pensions - they're all one one list, alphabetized by the soldier's name. See National Archives Film Series T-288, "Combined List of U.S. Pension Applications 1865 - 1931." In the LDS locality catalog, look under "United States - Military Records - Civil War, 1861-1865 - Indexes." Note that, despite the title, it's really the Civil War pension list - Indian fighters and Spanish American War vets have their own lists.
The index card is a little fancier here - it will mention the state where the application was filed, and give the name of the widow or orphan or parent who filed after the soldier was deceased. If the wife outlived the soldier, this makes it a lot easier to identify John and Matilda Smith than to pick old John alone out of the crowd.
Be sure to check with the individual state archives for additional indexes and information.