Phase II Trial of Nivolumab Plus Ipilimumab in Patients With Renal Medullary Carcinoma

Inclusion Criteria: * Patients must give written informed consent prior to initiation of therapy, in keeping with the policies of the institution. Patients with a history of major psychiatric illness must be judged able to fully understand the investigational nature of the study and the risks associated with the therapy. * Patients with locally advanced or metastatic RMC histologically confirmed by expert pathology review and loss of SMARCB1 staining by immunohistochemistry. Patients with advanced or metastatic unclassified renal cell carcinoma with medullary phenotype (a rare SMARCB1 negative RMC variant occurring in individuals without sickle hemoglobinopathies) are also eligible. The principal investigator (PI) is the final arbiter in questions related to eligibility. * Patients will be eligible regardless of whether they have had prior nephrectomy or still have their primary tumor in-situ. * Patients must have at least one measurable site of disease, defined as a lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) and measures \>= 15 mm with conventional techniques or \>= 10 mm with more sensitive techniques such as magnetic resonance imaging (MRI) or spiral computerized tomography (CT) scan. If the patient has had previous radiation to the marker lesion(s), there must be evidence of progression since the radiation. * Patients should be willing to provide a newly obtained fresh core biopsy of a tumor lesion. Not required if there is a recently obtained fresh specimen on an Institutional Review Board (IRB) approved correlated trial up to 6 weeks (42 days) prior to initiation of treatment on day 1. * Patients can be either naive for any previous systemic treatment or have had any number of prior systemic therapies. However, patients must not have received prior anticancer therapy with anti-PD1, anti-PD-L1, or anti-CTLA-4 immune checkpoint inhibitors. * There must be evidence of progression on or after last treatment regimen received. NOTE: If subject is unable to walk due to paralysis, but is mobile in a wheelchair, subject is considered to be ambulatory for the purpose of assessing their performance status. * Eastern Cooperative Oncology Group (ECOG) performance status 0-2. * Consent to MD Anderson companion laboratory protocol 2014-0938. * Hemoglobin \>= 9 g/dl (treatment allowed) (within 14 days of the first dose of the study drugs). * Absolute neutrophil count \>= 1000/mcL (within 14 days of the first dose of the study drugs). * Platelets \>= 75,000/L (within 14 days of the first dose of the study drugs). * Total bilirubin =\< 1.5 mg/dl (within 14 days of the first dose of the study drugs). * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) or alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X institutional upper limit of normal (ULN), except in known hepatic metastasis, wherein may be =\< 5 x ULN (within 14 days of the first dose of the study drugs). * Serum creatinine =\< 1.5 x ULN by gender (as long as patient does not require dialysis) (within 14 days of the first dose of the study drugs); May receive transfusion; Without growth factor support (filgrastim or pegfilgrastim) for at least 14 days; If creatinine is not \< 1.5 x ULN, then calculate by Cockcroft-Gault methods or local institutional standard and creatinine clearance (CrCl) must be \> 30 mL/kg/1.73 m\^2. * International normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x ULN prior to study entry. Therapeutic anticoagulation with warfarin is allowed if target INR =\< 3 on a stable dose of warfarin or on a stable dose of low molecular weight (LMW) heparin for \> 2 weeks (14 days) at the time of enrollment. * Patients with controlled brain metastases are allowed on protocol if they had solitary brain metastases that was surgically resected or treated with radiosurgery or Gamma knife, without recurrence or edema for 1 month (4 weeks). * Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of the study drug. * Women must not be breastfeeding. * WOCBP must agree to follow instructions for method(s) of contraception from the time of enrollment for the duration of treatment with study drug (s) plus 5 half-lives of study drug (s) plus 30 days (duration of ovulatory cycle) for a total of 5 months post treatment completion. * Men who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception for the duration of treatment with study drug (s) plus 5 half-lives of study drug (s) plus 90 days duration of sperm turnover for a total of 5 months post-treatment completion. * Azoospermic males and WOCBP who are continuously not heterosexually active are exempt from contraceptive requirements. However WOCBP must still undergo pregnancy testing as described in these sections. Exclusion Criteria: * Patients must not have any other malignancies within the past 2 years except for in situ carcinoma of any site, or adequately treated (without recurrence post-resection or post-radiotherapy) carcinoma of the cervix or basal or squamous cell carcinomas of the skin. * Patients currently receiving anticancer therapies or who have received anticancer therapies (including chemotherapy and targeted therapy) within 2 weeks (14 days) prior to study day are excluded. Patients who have completed palliative radiation therapy more than 14 days prior to the first dose of the combination ipilimumab plus nivolumab are eligible. * Patients, who have had a major surgery or significant traumatic injury (injury requiring \> 4 weeks \[28 days\] to heal) within 4 weeks (28 days) of start of study drug, patients who have not recovered from the side effects of any major surgery (defined as requiring general anesthesia) or patients that are expected to require major surgery, other than cytoreductive nephrectomy +/- retroperitoneal lymph node dissection, during the course of the study. * Patients who have organ allografts. * Known or suspected autoimmune disease. Patients with a history of inflammatory bowel disease (including Crohn's disease and ulcerative colitis) and autoimmune disorders such as rheumatoid arthritis, systemic progressive sclerosis (scleroderma), systemic lupus erythematosus or autoimmune vasculitis (e.g., Wegener's granulomatosis) are excluded from this study. Patients with a history of Hashimoto's thyroiditis only requiring hormone replacement, type I diabetes, or psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are allowed to participate. * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS). * Positive test for hepatitis B virus (HBV) using HBV surface antigen (HBVsAg) test or positive test for hepatitis C virus (HCV) using HCV ribonucleic acid (RNA) or HCV antibody test indicating acute or chronic infection. If hepatitis C antibody test is positive then active infection has to be confirmed by hepatitis C RNA testing for the patient to be excluded. * Any underlying medical condition, which in the opinion of the investigator, will make the administration of study drug hazardous or obscure the interpretation of adverse events, such as a condition associated with frequent diarrhea, uncontrolled nausea or vomiting. * Patients must not have received prior anticancer therapy with anti-PD1, anti-PD-L1, or anti- CTLA-4 immune checkpoint inhibitors. * Patients receiving any concomitant systemic therapy for renal cell cancer are excluded. * Patients must not be scheduled to receive another experimental drug while on this study. * Patients who are on high dose steroid (e.g., \> 10 mg prednisone daily or equivalent) or other more potent immune suppression medications (e.g., infliximab). Topical, inhaled, intraarticular, ocular, or intranasal corticosteroids (with minimal systemic absorption) are allowed. A brief course (\< 48 hours) of systemic corticosteroids for prophylaxis (e.g., from contrast dye allergy) is permitted. Physiological corticosteroid replacement therapy for adrenal insufficiency is also permitted. * Patients who have any severe and/or uncontrolled medical conditions or other conditions that could affect their participation in the study such as: a) Symptomatic congestive heart failure of New York heart Association class III or IV; b) Unstable angina pectoris, symptomatic congestive heart failure, myocardial infarction within 6 months of start of study drug, serious uncontrolled cardiac arrhythmia or any other clinically significant cardiac disease; c) Severely impaired lung function as defined as oxygen (O2) saturation that is 92% or less at rest on room air; d) Uncontrolled diabetes as defined by fasting serum glucose \> 1.5 x ULN; e) Systemic fungal, bacterial, viral, or other infection that is not controlled (defined as exhibiting ongoing signs/symptoms related to the infection and without improvement) despite appropriate antibiotics or other treatment; f) Known active or symptomatic viral hepatitis or chronic liver disease. Uncontrolled adrenal insufficiency. * Patients must not have history of other diseases, metabolic dysfunction, physical examination finding, or clinical laboratory finding giving reasonable suspicion of a disease or condition that contraindicates the use of ipilimumab or nivolumab or that might affect the interpretation of the results of the study or render the subject at high risk from treatment complications. * Patients should not receive immunization with attenuated live vaccines within one week (7 days) of study entry or during study period; Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed. * Uncontrolled brain or leptomeningeal metastases, including patients who continue to require glucocorticoids for brain or leptomeningeal metastases. * Female patients who are pregnant or breast feeding, or adults of reproductive potential who are not using effective birth control methods as defined above. If barrier contraceptives are being used, these must be continued throughout the trial by both sexes. Hormonal contraceptives are not acceptable as a sole method of contraception. * Any patients who cannot be compliant with the appointments required in this protocol must not be enrolled in this study.