Pembrolizumab in Treating Patients With Small Bowel Adenocarcinoma That is Metastatic or Locally Advanced and Cannot Be Removed by Surgery

Inclusion Criteria: * Patients must have biopsy-proven adenocarcinoma of the small bowel at any site (duodenum, jejunum, ileum), excluding ampullary and appendiceal tumors * Have locally advanced (unresectable) or metastatic small bowel adenocarcinoma * Willing and able to provide written informed consent for the trial * Measurable disease based on Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 * Had at least one prior line of systemic chemotherapy for metastatic disease; adjuvant therapy would not count toward first-line therapy unless patient recurs less than 6 months after completion of that regimen * Willing to provide blood and tissue (can be archival) samples for mandatory research purposes * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (1.50 x 10\^9 /L) obtained =\< 28 days prior to registration * Platelet count \>= 100,000/mm\^3 (100 x 10\^9 /L) obtained =\< 28 days prior to registration * Hemoglobin \>= 9.0 g/dL (5.6 mmol/L or 90 g/L) without transfusion or erythropoietin (EPO) dependency (within 7 days of assessment) obtained =\< 28 days prior to registration * Serum total bilirubin =\< 1.5 x upper limit of normal (ULN) OR direct bilirubin =\< ULN for subjects with total bilirubin levels \> 1.5 x ULN obtained =\< 28 days prior to registration * Aspartate transaminase (AST) and alanine aminotransferase (ALT) =\< 2.5 x ULN OR =\< 5 x ULN for subjects with liver metastases obtained =\< 28 days prior to registration * Serum creatinine =\< 1.5 x upper limit of normal (ULN) OR measured or calculated creatinine clearance must be \>= 60 mL/min for subjects with creatinine levels \> 1.5 X ULN using the Cockcroft-Gault formula (glomerular filtration rate \[GFR\] \>= 60 mL/min \[1.0 mL/s/m\^2\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) obtained =\< 28 days prior to registration * Female subject of childbearing potential have a negative urine or serum pregnancy =\< 7 days prior to registration; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Note: female subjects of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year; male subjects should agree to use an adequate method of contraception starting with the first dose of study therapy through 120 days after the last dose of study therapy Exclusion Criteria: * Non-adenocarcinoma histology * Adenocarcinoma originating in the ampulla or appendix; (duodenal tumors that involve the ampulla but originate in the duodenum are eligible) * Currently participating and receiving study therapy, or have participated in a study of an investigational agent and received study therapy, or used an investigational device =\< 4 weeks of registration * Diagnosis of immunodeficiency or be receiving systemic steroid therapy or any other form of immunosuppressive therapy =\< 7 days prior to registration * History of active TB (bacillus tuberculosis) * Hypersensitivity to pembrolizumab or any of its excipients * Any of the following: * Prior anti-cancer monoclonal antibody (mAb) =\< 4 weeks prior to registration * Prior chemotherapy, targeted small molecule therapy, or radiation therapy =\< 2 weeks prior to registration or who has not recovered to =\< grade 1 or baseline from adverse events due to the previously administered agent * Note: subjects with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Received major surgery =\< 2 weeks prior to registration, subject must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Known additional malignancy that is progressing or requires active treatment or that may interfere with interpretation of response evaluation, in the judgment of the investigator * Known active central nervous system (CNS) metastases and/or carcinomatous meningitis; subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging =\< 4 weeks prior to registration and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids =\< 7 days prior to registration; this exception does not include carcinomatous meningitis which is excluded regardless of clinical stability * Active autoimmune disease (including but not limited to: patients with a history of inflammatory bowel disease, including ulcerative colitis and Crohn?s disease, patients with a history of symptomatic disease \[e.g., rheumatoid arthritis, systemic progressive sclerosis (scleroderma), systemic lupus erythematosus, autoimmune vasculitis (e.g. Wegener?s granulomatosis)\]; CNS or motor neuropathy considered of autoimmune origin \[e.g., Guillain-Barre syndrome and myasthenia gravis, multiple sclerosis\]) that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs); replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * Known history of or any evidence of active, non-infectious pneumonitis * Active infection requiring systemic therapy * History or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject?s participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator * Known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * Received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent, or other immune checkpoint inhibitor (e.g. anti-CTLA4) * History of human immunodeficiency virus (HIV) (HIV 1/2 antibodies) * Known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid RNA \[qualitative\] is detected) * Received a live vaccine within 30 days of planned start of study therapy * Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however, intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines and are not allowed