Ipilimumab and Sargramostim in Treating Patients With Metastatic Prostate Cancer

Inclusion Criteria: * Histologically confirmed prostate cancer * Metastatic disease * Progressive disease after prior androgen deprivation as defined by at least 1 of the following criteria: * Patients with measurable disease must have an increase of at least 20% in the sum of the longest diameter of target lesions OR the appearance of 1 or more new lesions * Patients with nonmeasurable disease must have a positive bone scan and a prostate-specific antigen (PSA) level of at least 5 ng/mL, which has risen on at least 2 successive occasions at least 2 weeks apart\* * At least 1 PSA level must be obtained at least 4 weeks after flutamide (6 weeks after bicalutamide or nilutamide) * Testosterone no greater than 50 ng/dL * Patients with no prior orchiectomy must continue luteinizing hormone-releasing hormone agonist therapy * No history or radiologic evidence of CNS metastases * Performance status - ECOG 0-2 * At least 12 weeks * Absolute neutrophil count greater than 1,500/mm\^3 * Platelet count at least 100,000/mm\^3 * Hemoglobin at least 8 g/dL * Bilirubin less than 1.5 times upper limit of normal (ULN) * SGOT and SGPT no greater than 2.5 times ULN * Creatinine no greater than 1.5 times ULN * No significant cardiovascular disease * No New York Heart Association class III or IV congestive heart failure * No active angina pectoris * No myocardial infarction within the past 6 months * Not pregnant or nursing * Fertile patients must use effective contraception prior to, during, and for 3 months after study participation * No history of autoimmune disease including, but not limited to, any of the following: * Autoimmune hemolytic anemia * Ulcerative and hemorrhagic colitis * Endocrine disorders (e.g., thyroiditis, hyperthyroidism, hypothyroidism of immune etiology, autoimmune hypophysitis/hypopituitarism, or adrenal insufficiency) * Sarcoid granuloma * Myasthenia gravis * Polymyositis * Guillain-Barre syndrome * Systemic lupus erythematosus * Rheumatoid arthritis * Inflammatory bowel disease * No other medical or psychiatric illness that would preclude study participation or giving informed consent * No other malignancy within the past 5 years except adequately treated basal cell or squamous cell skin cancer or stage I or II cancer currently in complete remission * No prior immunotherapy (e.g., vaccines or investigational) * No other concurrent colony-stimulating factors * No prior chemotherapy * No concurrent chemotherapy * See Disease Characteristics * At least 4 weeks since prior systemic corticosteroids * At least 4 weeks since other prior hormonal therapy, including megestrol and finasteride * No concurrent systemic steroid therapy except inhaled or topical steroids * No other concurrent hormonal therapy * Hormones administered for nondisease-related conditions (e.g., insulin for diabetes) allowed * At least 4 weeks since prior radiotherapy and recovered * More than 8 weeks since prior radiopharmaceuticals (e.g., strontium chloride Sr 89 and samarium Sm 153 lexidronam pentasodium) * Prior irradiation of a symptomatic lesion or one that may produce disability (e.g., unstable femur) allowed * No concurrent palliative radiotherapy * See Disease Characteristics * At least 4 weeks since prior herbal products known to decrease PSA levels (e.g., PC-SPES or saw palmetto)