Rintatolimod and Pembrolizumab for the Treatment of Refractory Metastatic or Unresectable Colorectal Cancer

Inclusion Criteria: * Biopsy proven colorectal adenocarcinoma which is metastatic or otherwise unresectable * Microsatellite stable/mismatch-repair proficient (by immunohistochemistry \[IHC\] and/or polymerase chain reaction \[PCR\]) * Progression following: a fluoropyrimidine, oxaliplatin, irinotecan, and anti-EGFR targeted therapy (if anti-EGFR therapy is appropriate), bevacizumab (if appropriate) * NOTE: Patients who could not tolerate standard agents because of unacceptable, but reversible, toxicity necessitating their discontinuation will be allowed to participate * Amenable to undergoing serial tumor biopsy (x 2). NOTE: patients with inaccessible lesions, where the investigator deems biopsy to be unsafe or where biopsy is otherwise contraindicated, are still eligible to enroll, with review and approval of the principal investigator (PI) * Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Absolute neutrophil (ANC) count \>= 1500/uL * Platelets \>= 100,000/uL * Hemoglobin \>= 9 g/dL * Serum or plasma creatinine =\< 1.5 X upper limit of normal (ULN) or measured or calculated creatinine clearance by Cockcroft Gault Equation \>= 30 ml/min for subjects with creatinine levels \> 1.5 x ULN * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN or (=\< 5 x ULN if the patient has liver metastases) * Bilirubin =\< 1.5 x ULN OR direct bilirubin =\< ULN for participants with total bilirubin levels \> 1.5 x ULN * International normalized ratio (INR) or prothrombin time (PT): =\< 1.5 unless participant is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants * Activated partial thromboplastin time (aPTT): =\< 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants * Have measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria present * Female participants of childbearing potential are to have a negative serum pregnancy test * A female participant is eligible to participate if she is not pregnant, not breastfeeding, and at least one of the following conditions applies: * Not a woman of childbearing potential (WOCBP) OR * A WOCBP who agrees to follow the contraceptive guidance during the treatment period and for at least 120 days after the last dose of study treatment * A male participant must agree to use an adequate method of contraception during the treatment period and for at least 120 days after the last dose of study treatment and refrain from donating sperm during this period * Participant or legal representative must understand the investigational nature of this study and sign an Independent Ethics Committee/Institutional Review Board approved written informed consent form prior to receiving any study related procedure Exclusion Criteria: * Has received prior systemic anti-cancer therapy including investigational agents within 4 weeks prior to start of study treatment * Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette?Guerin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g., FluMist) are live attenuated vaccines and are not allowed * Has received prior radiotherapy within 2 weeks of start of study treatment. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. A 1-week washout is permitted for palliative radiation (=\< 2 weeks of radiotherapy) to non-central nervous system (CNS) disease * Is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of systemic immunosuppressive therapy within 7 days prior to the first dose of study drug * Has a known additional malignancy that is progressing or in the opinion of the investigator is likely to interfere with properly assessing treatment efficacy. Note: Participants with basal cell carcinoma of the skin, squamous cell carcinoma of the skin, and carcinoma in situ (e.g. breast carcinoma, cervical cancer in situ) that have undergone potentially curative therapy are not excluded * Has known active CNS metastases and/or carcinomatous meningitis. Participants with previously treated brain metastases may participate provided they are radiologically stable, i.e., without evidence of progression for at least 4 weeks by repeat imaging (note that the repeat imaging should be performed during study screening), clinically stable and without requirement of steroid treatment for at least 14 days prior to first dose of study treatment * Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, unstable cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Have a severe hypersensitivity (\>= grade 3) to pembrolizumab and/or any of its excipients * Has previously received rintatolimod, poly-ICLC or derivatives * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e., with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * Woman of childbearing potential who has a positive urine pregnancy test within 72 hours prior to start of study treatment. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Has an active infection requiring systemic therapy * Has a known history of human immunodeficiency virus (HIV), unless on highly active antiretroviral therapy (HAART) with undetectable viral load * Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) infection * Pregnant or nursing female participants * Unwilling or unable to follow protocol requirements * Any condition which in the investigator?s opinion deems the participant an unsuitable candidate to receive study drug