Nivolumab and Ipilimumab in Treating Patients With Rare Tumors

Inclusion Criteria: * Patients are eligible under ONE of the following criteria: * For all cohorts except the gestational trophoblastic disease (GTD) (Cohort #47), patients must have histologically and/or biochemically confirmed rare cancer and must be able to submit specimens; to be eligible for the GTD cohort: patients must have disease confirmed by quantitative serum beta-human chorionic gonadotropin (hCG) within 28 days prior to registration and must be able to submit blood specimens (tissue submission is not required for patients who will be registered to the GTD cohort \[Cohort #47\]); NOTE: Subsequent to site's Institutional Review Board (IRB) approval of revision 3, patients are NOT required to participate in EAY131 "National Cancer Institute (NCI)-Molecular Analysis for Therapy Choice (MATCH)" to register to S1609 OR * FOR PATIENTS WITH PD-L1 AMPLIFICATION (COHORT #50) ONLY: All solid tumors (excluding lymphoma) are allowed for the PD-L1 amplified cohort if they have PD-L1 amplification; patients may be considered for registration to the PD-L1 amplified cohort (Cohort #50) with the confirmation of at least one of the study chairs; PD-L1 amplification is defined as having deoxyribonucleic acid (DNA) copy number of equal to or greater than six by any of the following Clinical Laboratory Improvement Act (CLIA)-approved lab; (Immunohisochemistry \[IHC\] and fluorescence in situ hybridization \[FISH\] are not allowed); the assay must be done at or after the diagnosis of advanced disease, but PRIOR TO REGISTRATION; NOTE: patients with PD-L1 overexpression by IHC or PD-L1 amplification by FISH do not quality for this cohort; OR * FOR PATIENTS ENROLLED IN EAY131 "NCI-MATCH" PRIOR TO EAY131 ADDENDUM 10 ONLY: Patients must have histologically confirmed rare cancer that did not have a match to a molecularly-guided therapy on EAY131 "NCI-MATCH" protocol or who are off protocol treatment on EAY131, "NCI-MATCH" and have no further molecularly-matched treatment recommendations per EAY131, "NCI-MATCH" or who are otherwise unable to receive EAY131, "NCI-MATCH" therapy * Patients who do not qualify for one of the histologic cohorts and are not on the ineligible histology list may be considered for registration in the "Not Otherwise Categorized" Rare Tumors cohort with confirmation of at least one of the study chairs via email * NOTE: The "Not Otherwise Categorized" Rare Tumors cohort was permanently closed to accrual on 3/15/2019 * Patients who are determined to have a rare cancer with unknown primary site are eligible under cohort #32 (tumor of unknown primary \[cancer of unknown primary; CuP\]), provided that there is histologic documentation of metastatic malignancy with no discernible primary site identified from histopathologic review, physical exam and associated cross-sectional imaging of the chest, abdomen, and pelvis * NOTE: The "Tumor of unknown primary (Cancer of Unknown Primary; CuP" cohort was permanently closed to accrual on 12/22/2017 * Patients must also meet one of the following: * Patients must have progressed following at least one line of standard systemic therapy and there must not be other approved/standard therapy available that has been shown to prolong overall survival (i.e. in a randomized trial against another standard treatment or by comparison to historical controls); patients who cannot receive other standard therapy that has been shown to prolonged survival due to medical issues will be eligible, if other eligibility criteria are met; OR * Patients for whose disease no standard treatment exists that has been shown to prolong overall survival * For all cohorts except the GTD cohort (Cohort #47): Patients must have a diagnostic quality computed tomography (CT) scan or magnetic resonance imaging (MRI), performed within 28 days prior to registration, which demonstrates measurable disease, as defined in RECIST v. 1.1; scans must include imaging of the chest, abdomen and pelvis, with the exception of patients with head/neck cancer, who must have imaging of the chest, abdomen, pelvis and neck; if there is clinical suspicion for bone metastases at the time of enrollment (in the judgement of the treating investigator) bone scan should be performed; bone scans done within 42 days prior to registration may be used to establish baseline condition at registration * No other prior malignancy is allowed except for the following: * Adequately managed stage I or II cancer from which the patient is currently in complete remission * Any other cancer from which the patient has been disease free for one year * Adequately managed stage I or II follicular thyroid or prostate cancer is also eligible, wherein patient is not required to be in complete remission * Note: Second primary tumors are not allowed concurrent with any of the eligible rare cancers * For all cohorts except the PD-L1 amplified tumors cohort (Cohort # 50): Patients may have received either prior anti-CTLA4 or other prior anti-PD-1/anti-PD-L1 therapy, but not both, provided that it is completed \>= 4 weeks prior to registration. To be eligible for the PD-L1 amplified tumors cohort (Cohort #50): Patients must not have received anti-PD-1/anti-PD-L1 therapy; prior anti-CTLA-4 is allowed provided that it is completed \>= 4 weeks prior to registration * Patients who had prior grade 3 or higher immune-related adverse event (e.g. pneumonitis, hepatitis, colitis, endocrinopathy) with prior immunotherapy (e.g. cancer vaccine, cytokine, etc.) are not eligible * Patients with clinically controlled thyroiditis or pituitary disorders on stable replacement therapy are eligible * Patients are not eligible if they have had or are planned for solid organ transplant * Patients with autoimmune disease who are otherwise eligible must not have received steroid and immunosuppressive therapy within 28 days prior to registration * Patients with brain metastases or primary brain tumors must have completed treatment, surgery or radiation therapy \>= 28 days prior to registration and have stable disease at time of registration; these patients must also have a CT or MRI of the brain to evaluate for CNS disease within 42 days prior to registration to S1609; metastatic brain parenchymal disease must have been treated and patient must be off steroids for 7 days prior to registration * Patients must not currently be receiving any other investigational agents or any other systemic anti-cancer therapy (including radiation, excluding RANKL inhibitors and bisphosphonates); in event patient recently received any other systemic anti-cancer therapy, patient must be off therapy at least 7 days prior to registration and any therapy-induced toxicity must have recovered to =\< grade 1, except alopecia and =\< grade 2 neuropathy which are allowed; any planned radiation therapy must be completed before registration to S1609 * Patients must not have prior history of allergy or known hypersensitivity to nivolumab or ipilimumab * Hormonal or endocrine blockade is permitted as long as patient has demonstrated progression on prior therapy (e.g. gonadotrophin releasing hormone \[GnRH\], somatostatin); long-acting somatostatin analogs (including octreotide) and androgen deprivation treatment (including long-acting leuprolide) are permitted while on protocol therapy * Patients must be \>= 18 years of age * Patients must have a Zubrod performance status of 0-2 * Absolute neutrophil count (ANC) \>= 1,000/mcL (within 28 days prior to registration) * Platelets \>= 75,000/mcL (within 28 days prior to registration) * Hemoglobin \>= 8 g/dL (within 28 days prior to registration) * Total bilirubin =\< 2.0 x institutional upper limit of normal (IULN) or for documented/suspected Gilbert's disease, total bilirubin =\< 3.0 x IULN (within 28 days prior to registration) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) both =\< 3 x IULN (within 28 days prior to registration) * Serum creatinine =\< 2.0 x IULN (within 28 days prior to registration) * Creatinine clearance (CrCl) \>= 50 mL/min., as estimated by the Cockcroft and Gault formula; estimated creatinine clearance is based on actual body weight (within 28 days prior to registration) * Patients must have adequate thyroid function, as evidenced by either thyroid-stimulating hormone (TSH) or, free thyroxine (T4) serum tests demonstrating values within the normal range, within 28 days prior to registration; at pre-registration, if TSH is not within normal limits, then free T4 must be performed and must be within normal range for patient to be eligible; Note: TSH, with reflex T4 (if TSH is abnormal) is allowable if per institutional standard, provided that free T4 is within normal range; patients who have undergone thyroidectomy or who are on thyroid suppression for their cancer are not required to have normal TSH and free T4 * Patients must have adequate adrenal axis function, as evidenced by cortisol levels within institutional normal ranges (ante meridiem \[AM\] cortisol preferred), OR adrenocorticotropic hormone (ACTH) values within the institutional normal ranges within 28 days prior to registration; if cortisol levels are not within normal limits prior to registration, then ACTH must be performed and must be within normal ranges for patient to be eligible; Note: Neither cortisol nor ACTH levels are required for patients with primary adrenal tumors (e.g. adrenocortical carcinoma) * For women of childbearing potential, the local investigator must rule out pregnancy; Except for Cohorts 13 and 47, where tumor types may express beta-hCG, women of childbearing potential must have a serum or urine pregnancy test within 7 days prior to registration; for Cohorts 13 and 47, where tumor types may produce hCG (e.g. germ cell tumors or trophoblastic disease), other pregnancy exclusion methods should be used to rule out pregnancy, such as ultrasound examination, documented history of effective contraception, or documented infertility; all females of childbearing potential must have been demonstrated not to be pregnant within 7 days prior to registration and agree to use birth control throughout study and for 23 weeks after completion of protocol therapy; patients must not be pregnant or nursing due to risk of fetal or nursing infant harm; women of childbearing potential must have agreed to use an effective contraceptive method; a woman is considered to be of "childbearing potential" if she has had menses at any time in the preceding 12 consecutive months; in addition to routine contraceptive methods, "effective contraception" also includes heterosexual celibacy and surgery intended to prevent pregnancy (or with a side-effect of pregnancy prevention) defined as a hysterectomy, bilateral oophorectomy or bilateral tubal ligation; however, if at any point a previously celibate patient chooses to become heterosexually active during the time period for use of contraceptive measures outlined in the protocol, she is responsible for beginning contraceptive measures * Men of reproductive potential must have agreed to use birth control throughout the study and for 31 weeks after completion of protocol therapy; in addition to routine contraceptive methods, "effective contraception" also includes heterosexual celibacy and surgery intended to prevent pregnancy (vasectomy); however, if at any point a previously celibate patient chooses to become heterosexually active during the time period for use of contraceptive measures outlined in the protocol, he is responsible for beginning contraceptive measures * Patients must not have known active hepatitis B virus (HBV) or hepatitis virus (HCV) infection at time of registration; patients with HBV or HCV that have an undetectable viral load and no residual hepatic impairment are eligible * Patients who are known to be human immunodeficiency virus (HIV)-positive at registration are eligible at the time of registration: * CD4+ cell count greater or equal to 250 cells/mm\^3 * No history of non-malignancy acquired immunodeficiency syndrome (AIDS)-defining conditions other than historical low CD4+ cell counts * Patients must not have active autoimmune disease that has required systemic treatment in past 2 years (i.e., with use of disease modifying agents, immunosuppressive drugs, or corticosteroids with prednisone dose \>= 10 mg); replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment; autoimmune diseases include but are not limited to autoimmune hepatitis, inflammatory bowel disease (including ulcerative colitis and Crohn's disease), as well as symptomatic disease (e.g.: rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, autoimmune vasculitis \[e.g., Wegener's granulomatosis\]); central nervous system (CNS) or motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre syndrome and Myasthenia gravis, multiple sclerosis or glomerulonephritis); vitiligo, alopecia, hypothyroidism on stable doses of thyroid replacement therapy, psoriasis not requiring systemic therapy within the past 2 years is permitted; short-term steroid premedication for contrast allergy is permitted * Patients must not have any uncontrolled intercurrent illness including (not limited to): symptomatic congestive heart failure (CHF) (New York Heart Association \[NYHA\] III/IV), unstable angina pectoris or coronary angioplasty, or stenting within 24 weeks prior to registration, unstable cardiac arrhythmia (ongoing cardiac dysrhythmias of NCI Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\] 4 grade \>= 2), known psychiatric illness that would limit study compliance, intra-cardiac defibrillators, known cardiac metastases, or abnormal cardiac valve morphology (\>= grade 3) * Note: Patients with history of CHF or patients who are deemed at risk because of underlying cardiovascular disease or exposure to cardiotoxic drugs should have an electrocardiogram (EKG) and echocardiogram (ECHO), as clinically indicated, at baseline and at the start of each cycle; patients who have evidence at baseline (or subsequently) of CHF, myocardial infarction (MI), cardiomyopathy, or myositis cardiac evaluation (NYHA I/II) should have additional consult by a cardiologist, including review of EKG, creatine phosphokinase (CPK), troponin, echocardiogram, as clinically indicated * Patients must have amylase or lipase within =\< 1.5 x IULN without symptoms of pancreatitis at registration, within 28 days prior to registration * Patients must not have symptomatic interstitial lung disease or pneumonitis * Patients must have fully recovered from any adverse effects of major surgery (to =\< grade 1) at least 14 days prior to registration