Antiandrogen Therapy, Abiraterone Acetate, and Prednisone With or Without Neutron Radiation Therapy in Treating Patients With Prostate Cancer

Inclusion Criteria: * Pathologically proven (either histologic or cytologic) diagnosis of prostate adenocarcinoma with \< 50% neuroendocrine differentiation or small cell histology. * At least one site of nodal or distant metastatic disease that is measurable by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria, or a bony metastasis that is evaluable on both computed tomography (CT) and bone scan. * No prior orchiectomy. * No androgen deprivation therapy such as treatment with antiandrogens, luteinizing hormone-releasing hormone (LHRH) agonists or antagonists for at least one year prior to trial enrollment, and testosterone must be inside normal range prior to trial enrollment if there is prior history of ADT. * No other systemic anti-cancer therapy for at least 1-year prior to enrollment. * Prior prostate-directed therapies such as prostatectomy or cryotherapy are allowed. * Prior radiation treatments are allowed (prostate or metastatic sites) but must have been completed at least 3 months prior to starting ADT for this trial. * White blood cell (WBC) \> 3000/mm\^3. * Absolute neutrophil count (ANC) \> 1000/mm\^3. * Platelets \> 100,000/mm\^3. * Creatinine \< 1.5 institutional upper limit of normal (ULN) or calculated creatinine clearance \> 30 ml/min. * Aspartate aminotransferase (AST), alanine aminotransferase (ALT), and total bilirubin \< 3 x institutional ULN (unless patient has documented Gilbert's syndrome). * No steroids for at least 2 weeks prior to enrollment, and patient must not be expected to require steroids during the study period, other than the typical low dose steroid that is given with abiraterone (typically prednisone or prednisolone at 5 mg twice daily). * Zubrod performance status 0-2. * Patient must sign study specific informed consent prior to study entry. * Men who are sexually active must use medically acceptable forms of contraception. Exclusion Criteria: * Other illnesses with a life expectancy of less than 6 months, including but not limited to unstable angina, symptomatic congestive heart failure, cardiac arrhythmias. * Psychological or social issues that would prevent patients from informed consent or complying with study requirements. * Subject has a history of unexplained loss of consciousness or transient ischemic attack within 12 months of treatment start. * Individuals on active treatment for a different cancer are excluded. Individuals with a history of other malignancies are eligible if they are deemed by the investigator to be at low risk for recurrence of that malignancy. * Known brain metastasis. * Known allergies, hypersensitivity, or intolerance to abiraterone or prednisone. * Prior ADT less than a year, or greater than two months, prior to trial enrollment or prior ADT with testosterone less than normal. * There is a potential drug interaction when abiraterone is concomitantly used with a CYP2D6 substrate narrow therapeutic index (e.g., thioridazine, dextromethorphan), or strong CYP3A4 inhibitors (e.g., atazanavir, erythromycin, indinavir, itraconazole, ketoconazole, nefazodone, nelfinavir, ritonavir, saquinavir, telithromycin, and voriconazole) or strong inducers (e.g., carbamazepine, phenobarbital, phenytoin, rifabutin, rifampin, rifapentine). Caution should be used when patients are on one of these drugs. * Patients with a history of pituitary or adrenal dysfunction, active or symptomatic viral hepatitis, human immunodeficiency virus (HIV), or chronic liver disease are not eligible. * Any chronic medical condition requiring a higher dose of corticosteroid than 10 mg prednisone/prednisolone once daily.