IRX-2, Cyclophosphamide, and Nivolumab in Treating Patients With Recurrent or Metastatic and Refractory Liver Cancer

Inclusion Criteria: * Patients with histologically or cytologically confirmed recurrent or metastatic hepatocellular carcinoma (HCC). * Patients must have recurrent or metastatic HCC that are not amenable to local therapy with curative intent (surgery or radiation therapy with or without chemotherapy). * Must have failed or not tolerated at least one line of treatment for advanced HCC. * Willing and able to give informed consent and adhere to protocol therapy; written informed consent and any locally required authorization must be obtained from the patient prior to performing any protocol?related procedures, including screening evaluations. * Up to three prior systemic therapy regimens for recurrent and/or metastatic disease. * Eastern Cooperative Oncology Group (ECOG) 0?1. * Have a Child?Pugh class A liver score within 7 days of first dose of study drug. * Hemoglobin \> 8 g/dL. * Absolute neutrophil count (ANC) \> 1,200 x 10\^9/mL. * Platelet count \> 60 x 10\^9/mL. * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). * Aspartate aminotransferase (AST/serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT/serum glutamate pyruvate transaminase \[SGPT\]) =\< 5 x ULN. * Serum albumin \> 3.0 g/dL. * Prothrombin time (PT) and partial thromboplastin time (PTT) \< 1.5 x the ULN. * Measured creatinine clearance (CL) \> 40 mL/min or calculated creatinine clearance CL \> 40 mL/min by the Cockcroft?Gault formula 10 or by 24?hour urine collection for determination of creatinine clearance. * Palliative radiation therapy is allowed to non?target lesions at the discretion of the treating physician. * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as outlined in RECIST version 1.1. * Life expectancy of greater than 3 months. * Subjects with chronic infection by hepatitis C virus (HCV) who are untreated are allowed on study. In addition, subjects with successful HCV treatment (defined as sustained virologic response \[SVR\] 12 or SVR 24) are allowed as long as 4 weeks have passed between completion of HCV therapy and start of study drug. * Subjects with hepatitis B virus (HBV) may only be enrolled if their hepatitis is judged clinically stable by the investigator. * Female patients of childbearing potential must have a negative serum pregnancy test within 7 days prior to enrollment. * Body weight \> 30 Kg. Exclusion Criteria: * Prior exposure to PD?1/PD?L1 inhibitors. * Prior exposure to IRX?2 regimen. * Radiation therapy with a curable intent within 30 days of first dose of study treatment is excluded. However, radiation therapy with a palliative intent is allowed as long as treatment with study medication occurs \>= 14 days after last dose of radiation and as long as there is at least 1 evaluable non?treated target lesion remaining. * Any medical contraindications or previous therapy that would preclude treatment with the IRX 2 regimen or nivolumab including the following: * Patients with allergies to ciprofloxacin or phytohemagglutinin (PHA). * Patients with evidence of pre?existing myelosuppression, myelodysplasia or hemorrhagic cystitis. * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria. * Patients with grade \>= 2 neuropathy will be evaluated on a case?by?case basis after consultation with the study physician. * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with IRX?2, nivolumab may be included only after consultation with the study physician. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia. * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement. Any chronic skin condition that does not require systemic therapy. * Patients without active disease in the last 2 years may be included but only after consultation with the study physician. * Patients with celiac disease controlled by diet alone. * Current or prior use of immunosuppressive medication within 14 days before the first dose of nivolumab. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroid or local steroid injections (e.g., intra articular injection). * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent. * Steroids as premedication for hypersensitivity reactions (e.g., computed tomography \[CT\] scan premedication). * Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of nivolumab. * Note: Local surgery of isolated lesions for palliative intent is acceptable. * History of allogenic organ transplantation. * Symptomatic cardiopulmonary disease (including congestive heart failure and hypertension), coronary artery disease, serious arrhythmia or chronic lung disease. Patients with these conditions who are stable with relatively minor symptoms and who are appropriate candidates for systemic treatments need not be excluded. * Myocardial infarction within the last 3 months. * Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies). * Has untreated active Hepatitis B. * Note: To qualify for enrollment, antiviral therapy for HBV must be given for at least 3 months, and HBV viral load must be less than 100 IU/mL prior to first dose of study drug. Those on active HBV therapy with viral loads under 100 IU/mL should stay on the same therapy throughout trial treatment. Those subjects who are anti?HBc (+), and negative for hepatitis B surface antigen (HBsAg), and negative for anti? HBs, and have an HBV viral load under 100 IU/mL do not require HBV anti?viral prophylaxis, but need close monitoring. * Has dual infection with HBV/HCV or other hepatitis combinations at study entry. * Receipt of live attenuated vaccine within 4 months prior to the first dose of study treatment. * Note: Patients, if enrolled, should not receive live vaccine whilst receiving study treatment and up to 4 months after the last dose of study treatment. Inactivated vaccines should not be administered within 2 weeks prior to or after study regimen (ideally 4 weeks). * Signs or symptoms of systemic infection (use of antibiotics to treat superficial infection or contamination of tumor shall not, by itself, be considered evidence of infection). * Stroke or other symptoms of cerebral vascular insufficiency within the last 3 months. * Previous diagnosis of invasive cancer from which the individual is not disease?free AND that has required treatment within the past 3 years, except for superficial skin, cervical cancer in?situ, or early stage prostate or bladder cancer (i.e. treatment with curative intent and long term disease?free expectations). * History of leptomeningeal carcinomatosis. * Known allergy or hypersensitivity to any of the study drugs or any of the study drug excipients. * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to one year after last dose of cyclophosphamide or 180 days after the last dose of nivolumab therapy, whichever is longer.