Nivolumab in Treating Patients With Metastatic Adrenocortical Cancer

Inclusion Criteria: * Patients must have a histologically confirmed stage IV or unresectable locally advanced adrenocortical carcinoma * Patients must have disease progressing after treatment with at least one line of therapy including mitotane and/or chemotherapy; Note: patients declining first line treatment with mitotane and/or chemotherapy based on limited efficacy are also eligible for this study * Patients must have measurable disease according to the standard RECIST version 1.1; CT scans or magnetic resonance imaging (MRIs) used to assess the measurable disease must have been completed within 28 days prior to registration * Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) performance status of 0-3 * Leukocytes \>= 2,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 9 g/dL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 × institutional upper limit of normal (ULN) (except patients with Gilbert Syndrome, who can have total bilirubin \< 3.0 mg/dL) * Aspartate transaminase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine transferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN * Serum creatinine of \< 3.0 x ULN (upper limit of normal) or creatinine clearance (CrCl) \> 30 mL/minute (using Cockcroft/Gault formula below) * Patients with history of central nervous system (CNS) metastases are eligible if CNS disease has been radiographically and neurologically stable for at least 6 weeks prior to study registrations and do not require corticosteroids (of any dose) for symptomatic management * Females of childbearing potential (FOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 72 hours prior to the start of study drug; NOTE: a FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria: has not undergone a hysterectomy or bilateral oophorectomy; has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * FOCBP must agree to follow instructions for method(s) of contraception (e.g. hormonal or barrier method of birth control; abstinence) for the duration of treatment with nivolumab plus 5 half-lives of nivolumab (19 weeks) plus 30 days (duration of ovulatory cycle) for a total of 23 weeks post-treatment completion * Males who are sexually active with women of childbearing potential (WOCBP) must agree to follow instructions for method(s) of contraception (e.g. hormonal or barrier method of birth control; abstinence) for the duration of treatment with nivolumab plus 5 halflives of the study drug (19 weeks) plus 90 days (duration of sperm turnover) for a total of 31 weeks days post-treatment completion * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study Exclusion Criteria: * Patients who have had chemotherapy or radiotherapy within 4 weeks prior to entering the study are not eligible * Patients who have not recovered to =\< grade 1 from adverse events due to agents administered more than 4 weeks earlier are not eligible * Patients may not be receiving any other investigational agents * Patients who have a history of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab are not eligible * Patients should be excluded if they have had prior treatment with an anti-PD1 or anti-PD-L1. Please contact principal investigator, Benedito Carneiro, for specific questions on potential interactions * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including chronic prolonged systemic corticosteroids (defined as corticosteroid use of duration one month or greater), should be excluded; these include but are not limited to patients with a history of: * Immune related neurologic disease * Multiple sclerosis * Autoimmune (demyelinating) neuropathy * Guillain-Barre syndrome * Myasthenia gravis * Systemic autoimmune disease such as systemic lupus erythematosus (SLE) * Connective tissue diseases * Scleroderma * Inflammatory bowel disease (IBD) * Crohn's * Ulcerative colitis * Patients with a history of toxic epidermal necrolysis (TEN) * Stevens-Johnson syndrome * Anti-phospholipid syndrome; Note: subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * Any condition requiring systemic treatment with corticosteroids (\> 10mg daily prednisone equivalents) or other immunosuppressive medications within 14 days prior to first dose of study drug; Note: inhaled steroids and adrenal replacement steroid doses \> 10mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; a brief (less than 3 weeks) course of corticosteroids for prophylaxis (eg, contrast dye allergy) or for treatment of non-autoimmune conditions (eg, delayed-type hypersensitivity reaction caused by a contact allergen) is permitted * Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible * Hypertension that is not controlled on medication (Note: hypertension is defined as blood pressure \>= 140/90) * Ongoing or active infection requiring systemic treatment * Symptomatic congestive heart failure * Unstable angina pectoris * Cardiac arrhythmia * Psychiatric illness/social situations that would limit compliance with study requirements * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Female patients who are pregnant or nursing are not eligible * No other prior malignancy is allowed except for the following: adequately treated basal cell or squamous cell skin cancer, in situ cervical cancer, adequately treated stage I or II cancer from which the patient is currently in complete remission, or any other cancer from which the patient has been disease free for at least three years * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS) is not permitted * Any known positive test for hepatitis B or hepatitis C virus indicating acute or chronic infection is not permitted