Nivolumab With or Without Ipilimumab in Treating Patients With Gastrointestinal Stromal Tumor That Is Metastatic or Cannot Be Removed by Surgery

Inclusion Criteria: * Written informed consent must be obtained from the subject/legal representative prior to performing any protocol-related procedures, including screening evaluations * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1 * Life expectancy \> 12 weeks * Histological confirmation of GIST * Mutational testing of patient samples for KIT and platelet-derived growth factor receptor (PDGFR) mutations; (this will not hold up starting therapy, but will be done for all patients lacking up front mutational testing) * Patients must have refused or have evidence of intolerance to or progression on imatinib * This study permits the re-enrollment of a subject that has discontinued the study as a pre-treatment failure (i.e., subject has not been randomized / has not been treated); if re-enrolled, the subject must be re-consented * Adequate archival tissue must be available from the prior 3 months to signing consent; if not, an adequate tumor specimen obtained by either excisional biopsy, incisional biopsy or core needle biopsy must be sent to the central pathology lab for evaluation; the material must measure at least 0.8 × 0.1 cm in size or contain at least 100 tumor cells * Measurable tumor lesions according to Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria * Hemoglobin \>= 9 g/dL * Absolute neutrophil count \>= 1,500/mm\^3 * Platelet count \>= 100,000/mm\^3 * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 × institutional upper limit of normal (ULN); for subjects with liver metastases, AST or ALT =\< 5 × ULN * Bilirubin =\< 1.5 × ULN; for subjects with documented/suspected Gilbert's disease, bilirubin =\< 3 × ULN * Willingness to provide consent for biopsy samples; tumor biopsies will be required for all subjects, tumor lesions used for biopsy should not be lesions used as RECIST target lesions, unless there are no other lesions suitable for biopsy; if a RECIST target lesion is used for biopsy the lesion must be \>= 2 cm in longest diameter * Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of study drug * Women must not be breastfeeding * Women of childbearing potential (WOCBP) must agree to follow instructions for method(s) of contraception for the duration of treatment with study drug(s) plus 5 half-lives of study drug (25 days) plus 30 days (duration of ovulatory cycle) for a total of 155 days post-treatment completion * Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception for the duration of treatment with study drug(s) plus 5 half-lives of the study drug plus 90 days (duration of sperm turnover) for a total of 215 days post-treatment completion; in addition, male subjects must be willing to refrain from sperm donation during this time * Azoospermic males are exempt from contraceptive requirements; WOCBP who are continuously not heterosexually active are also exempt from contraceptive requirements, and still must undergo pregnancy testing as described in this section * Male subjects must be willing to refrain from sperm donation during the entire study and for 5 half-lives of study drug plus 90 days (duration of sperm turnover) after dosing has been completed * At a minimum, subjects must agree to use one highly effective (which have a failure rate of \< 1% when used consistently and correctly) OR one less effective method of contraception as listed below: * WOCBP and female partners of male subjects, who are WOCBP, are expected to use one of the highly effective methods of contraception listed below; male subjects must inform their female partners who are WOCBP of the contraceptive requirements of the protocol and are expected to adhere to using contraception with their partner; contraception methods are as follows: * Progestogen only hormonal contraception associated with inhibition of ovulation * Hormonal methods of contraception including oral contraceptive pills containing combined estrogen + progesterone, vaginal ring, injectables, implants and intrauterine devices (IUDs) such as Mirena * Nonhormonal IUDs, such as ParaGard * Bilateral tubal occlusion * Vasectomised partner with documented azoospermia 90 days after procedure * Intrauterine hormone-releasing system (IUS) * Complete abstinence * Less effective methods of contraception * Diaphragm with spermicide * Cervical cap with spermicide * Vaginal sponge with spermicide * Male or female condom with or without spermicide * Progestogen-only oral hormonal contraception, where inhibition of ovulation is not the primary mode of action * Unacceptable methods of contraception * Periodic abstinence (calendar, symptothermal, post-ovulation methods) * Withdrawal (coitus interruptus) * Spermicide only * Lactation amenorrhea method (LAM) Exclusion Criteria: * Palliative surgery and/or radiation treatment within 28 days prior to course 1 day 1 (C1D1) * Localized therapy of non-target lesions is allowed * No steroids are permitted within 28 days of C1D1; or doses \< 10mg/day prednisone equivalent or levels necessary for physiologic replacement * Women who are of pregnant or breastfeeding * Inability to give informed consent * An inadequate tumor specimen as defined by the local pathologist * History of other malignancies except cured basal cell carcinoma, cutaneous squamous cell carcinoma, melanoma in situ, superficial bladder cancer or carcinoma in situ of the cervix; for other malignancies, must be documented to be free of cancer for \>= 2 years; all other cases can be considered on a case by case basis at the discretion of the principal investigator * Any condition that might interfere with the subject's participation in the study, safety, or in the evaluation of the study results * Concurrent enrollment in another clinical study, unless it is an observational (non-interventional) clinical study or the follow-up period of an interventional study * Prior exposure to any anti-PD-1 or anti-PD-L1 antibody, or any anti-cytotoxic T-lymphocyte-associated protein (CTLA) 4 antibodies * Any concurrent chemotherapy, immunotherapy, biologic or hormonal therapy for cancer treatment; concurrent use of hormones for non-cancer-related conditions (eg, insulin for diabetes and hormone replacement therapy) is acceptable * Current or prior use of immunosuppressive medication within 28 days before the first dose of nivolumab, with the exceptions of intranasal, topical, and inhaled corticosteroids or systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or equivalent (use for brain metastases is not permitted 28 days prior to start of therapy) * Active or prior documented autoimmune disease within the past 3 years * NOTE: subjects with active, known or suspected autoimmune disease such as vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * History of organ transplant that requires use of immunosuppressives * Active or prior documented inflammatory bowel disease (e.g. Crohn's disease, ulcerative colitis) or a history of primary immunodeficiency * Exposure to any therapeutic agent (investigational or conventional) within 7 days of date of randomization or to any agent for which 5 half lives have not elapsed * Unresolved toxicities from prior anticancer therapy, defined as having not resolved to National Cancer Institute's (NCI's) Common Terminology Criteria for Adverse Events (CTCAE) (NCI CTCAE version \[v\]4.03) grade 0 or 1, or to levels dictated in the inclusion/exclusion criteria with the exception of alopecia; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by nivolumab may be included (eg, hearing loss) after consultation with the principal investigator * Subjects who active hepatitis B or C, or human immunodeficiency virus (HIV) * Prisoners or subjects who are involuntarily incarcerated; (Note: under certain specific circumstances a person who has been imprisoned may be included or permitted to continue as a subject * Subjects who are compulsorily detained for treatment of either a psychiatric or physical (eg, infectious disease) illness