Nivolumab With or Without Ipilimumab in Treating Patients With Refractory Metastatic Anal Canal Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed previously treated metastatic squamous cell carcinoma of the anal canal * Patients must have measurable disease according to the standard Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1. CT scans or MRIs used to assess the measurable disease must have been completed within 28 days prior to study drug initiation * Patients must have been treated with at least one prior systemic treatment for incurable advanced or metastatic SCCA of the anal canal; prior treatment for metastatic disease is not required for patients who develop new metastatic lesions during or within 6 months of completion of chemoradiation for limited-stage disease; patients who receive chemotherapy for incurable advanced or metastatic SCCA of the anal canal must wait a minimum \>= 28 days (6 weeks for nitrosoureas or mitomycin C) after the date of completion of chemotherapy prior to initiating treatment with nivolumab on this study; patients who undergo palliative radiotherapy to a site of tumor must wait a minimum \>= 28 days from the date of completion of radiotherapy prior to initiating treatment with nivolumab (Part A and B) or nivolumab +/- Ipilimumab (Part B) on this study * Patients must be of age \>= 18 years at the time of study registration; because no dosing or adverse event data are currently available on the use of nivolumab in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status 0 or 1 (Karnofsky \>= 80%) * Leukocytes \>= 2,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 9.0 gm/dL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (except patients with Gilbert syndrome, who can have total bilirubin \< 3.0 mg/dL) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN * Serum creatinine =\< 1.5 x ULN OR creatinine clearance (CrCl) \>= 50 mL/min (if using the Cockcroft-Gault formula) * Before study enrollment, women of child bearing potential must be advised of the importance of avoiding pregnancy during study participation and the potential risk factors for an unintentional pregnancy; the subject must sign an informed consent form documenting this discussion; the effects of nivolumab and ipilimumab on the developing human fetus are unknown; for this reason women of child-bearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; women of childbearing potential MUST have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of nivolumab with or without ipilimumab; the minimum sensitivity of the pregnancy test must be 25 IU/L or equivalent units of HCG; if the pregnancy test is positive, the subject must not receive nivolumab with or without ipilimumab and must not be enrolled in the study * Women of childbearing potential (WOCBP) is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal; menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes * WOCBP receiving nivolumab (Parts A+B) or nivolumab and ipilimumab (Part B) will be instructed to adhere to contraception for a period of 5 months after the last dose of investigational product; men receiving nivolumab (Parts A+B) or nivolumab and ipilimumab (Part B only) and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of investigational product; these durations have been calculated using the upper limit of the half-life for nivolumab (25 days) and are based on the protocol requirement that WOCBP use contraception for 5 half-lives plus 30 days and men who are sexually active with WOCBP use contraception for 5 half-lives plus 90 days * Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she (or the participating partner) should inform the treating physician immediately; if, following initiation of the investigational product, it is subsequently discovered that a study subject is pregnant or may have been pregnant at the time of investigational product exposure, including during at least 6 half-lives after product administration, the investigational product will be permanently discontinued in an appropriate manner (e.g., dose tapering if necessary for subject safety); the investigator must immediately notify BMS of this event and record the pregnancy on the Pregnancy Surveillance Form (not an severe adverse event \[SAE\] form); initial information on a pregnancy must be reported immediately to BMS, and the outcome information provided once the outcome is known; completed Pregnancy Surveillance Forms must be forwarded to BMS according to SAE reporting procedures; any pregnancy that occurs in a female partner of a male study participant should be reported to the sponsor. Information on this pregnancy will be collected on the Pregnancy Surveillance Form; protocol-required procedures for study discontinuation and follow-up must be performed on the subject unless contraindicated by pregnancy (e.g., X-ray studies); other appropriate pregnancy follow-up procedures should be considered if indicated; in addition, the investigator must report and follow-up on information regarding the course of the pregnancy, including perinatal and neonatal outcome; infants should be followed for a minimum of 8 weeks * Ability to understand and the willingness to sign a written informed consent document * Brain metastases are allowed if they have been adequately treated with radiotherapy or surgery and have been stable for at least three months prior to registration; eligible subjects should be neurologically asymptomatic; there is no magnetic resonance imaging (MRI) evidence of progression for a minimum of 4 weeks after treatment is complete and within 28 days prior to the first dose of nivolumab administration; there must also be no requirement for immunosuppressive doses of systemic corticosteroids (\> 10 mg/day prednisone equivalents) for at least 2 weeks prior to study drug administration * Willingness for evaluation of cardiac function including electrocardiography (EKG) and echocardiogram (ECHO) cardiogram for any patients with a history of congestive heart failure (CHF) or at risk because of underlying cardiovascular disease or exposure to cardiotoxic drugs as clinically indicated * All patients must be willing to undergo testing for human immunodeficiency virus (HIV) testing if not tested within the past 6 months * If HIV+ positive, all patients infected with human immunodeficiency virus (HIV) may be eligible for study provided that their CD4+ count \>= 300/uL; their viral load is undetectable; they are currently receiving highly active antiretroviral therapy (HAART) * All HIV+ patients will be under the care of an infectious diseases specialist; if a relationship with an infectious diseases specialist is not established, infectious disease specialist will be consulted; records of all viral counts and peripheral T-cell counts must be sent to the study coordinator in order to follow these values over the course of treatment * All patients must be willing to be tested for hepatitis screening; patients co-infected with hepatitis B virus and/or hepatitis C virus may be included in this study provided that their liver function tests remain within the limits listed above; patients must be followed by a hepatologist during the course of this study Exclusion Criteria: * Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events (AEs) due to agents administered more than 4 weeks earlier (i.e., grade \>= 2 AE present); palliative (limited-field) radiation therapy is permitted, as long as the lesion being considered for palliative radiation is not a target lesion * Patients who are receiving any other investigational agents * Patients should be excluded if they have had prior treatment with an anti-PD-1, anti-PD-L1, anti-programmed cell death ligand 2 (PD-L2), anti-cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint pathways * History of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab (Parts A+ B) and/or ipilimumab (Part B) * History of severe hypersensitivity reaction to any monoclonal antibody * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including chronic prolonged systemic corticosteroids (defined as corticosteroid use of duration one month or greater), should be excluded; these include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or anti-phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease * Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses =\< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption); physiologic replacement doses of systemic corticosteroids are permitted, even if =\< 10 mg/day prednisone equivalents; a brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted * No other prior malignancy is allowed except for the following: adequately treated basal cell or squamous cell skin cancer, in situ cervical cancer, adequately treated stage I or II cancer from which the patient is currently in complete remission, or any other cancer from which the patient has been disease free for at least three years