Regorafenib and Durvalumab for the Treatment of High-Risk Liver Cancer

Inclusion Criteria: * Age \>= 18 years at time of study entry * Body weight \> 30 kg * Patients must have hepatocellular carcinoma (HCC) diagnosis confirmed by histology/cytology or clinically by American Association for Study of liver Diseases (AASLD) criteria in cirrhotic patients * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0 or 1 * Child Pugh class A * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (obtained =\< 28 days prior to registration) * Alanine aminotransferase (ALT) =\< 5 x ULN (obtained =\< 28 days prior to registration) * Aspartate transaminase (AST) =\< 5 x ULN (obtained =\< 28 days prior to registration) * Prothrombin time (PT)/international normalized ratio (INR)/activated partial thromboplastin time (aPTT) =\< 1.5 x ULN OR if patient is receiving anticoagulant therapy and INR or aPTT is within target range of therapy (obtained =\< 28 days prior to registration) * Creatinine =\< 1.5 x ULN or creatinine clearance \>= 60 mL/min/1.73 m\^2 for patients with creatinine levels \> 1.5 x ULN. Creatinine clearance should be calculated per institutional standard (obtained =\< 28 days prior to registration) * Urinary protein is =\< 1+ on dipstick or routine urinalysis or 24-hour urine demonstrating \< 1 gram of protein (obtained =\< 28 days prior to registration) * Absolute neutrophil count (ANC) \>= 1.0 x 10\^9/Liter (obtained =\< 28 days prior to registration) * Hemoglobin \>= 9 grams/deciliter (obtained =\< 28 days prior to registration) * Platelets \>= 75 x 10\^9/Liter (obtained =\< 28 days prior to registration) * Patient is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up * Life expectancy of \>= 12 weeks * Negative pregnancy test done =\< 7 days prior to registration, for women of childbearing potential only * NOTE: If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Provide informed written consent =\< 28 days prior to registration * Willing to return to enrolling institution for follow-up (during the active monitoring phase of the study) * Note: During the active monitoring phase of a study (i.e., active treatment and clinical follow-up), participants must be willing to return to the consenting institution for follow-up * Willing to provide mandatory tissue specimens and/or blood specimens for correlative research purposes * Clinical staging T1b/T2 or T3 hepatocellular cancer TNM staging American Joint Committee on Cancer (AJCC) International Union Against Cancer (UICC) 8th edition * Solitary tumor \> 5 cm without vascular invasion * T2: Solitary tumor \> 2 cm with vascular invasion, or multiple tumors, none \> 5 cm * T3: Multiple tumors, at least one of which is \> 5 cm * Able to swallow oral medication Exclusion Criteria: * Prior therapy with anti-PD-1, PD L-1 antibody including durvalumab, regorafenib, or other approved systemic therapies for HCC * Mixed histology HCC or fibrolamellar HCC * History of upper gastrointestinal bleed =\< 6 months from registration * Liver directed therapy =\< 28 days prior to registration. Prior liver directed therapy \> 28 days prior to registration is allowed * Evidence of extrahepatic metastatic disease * Suitable for liver transplant * Participation in another clinical study where the patient has received any dose of an investigational product =\< 90 days prior to registration * Concurrent enrollment in another clinical study, unless it is an observational (non-interventional) clinical study or during the follow-up period of an interventional study * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab may be included only after consultation with the study physician * Any concurrent chemotherapy, investigational product (IP), biologic, or hormonal therapy for cancer treatment. Concurrent use of hormonal therapy for non-cancer-related conditions (e.g., hormone replacement therapy) is acceptable * Major surgical procedure (as defined by the investigator) =\< 28 days prior to registration * History of allogenic organ transplantation * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the study physician * Patients with celiac disease controlled by diet alone * Uncontrolled intercurrent illness, including but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirement, substantially increase risk of incurring adverse events (AEs) or compromise the ability of the patient to give written informed consent * History of another primary malignancy except for * Malignancy treated with curative intent and with no known active disease \>= 3 years before registration and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease * History of leptomeningeal carcinomatosis * Single 12 Lead electrocardiogram (ECG) which Fridericia's correction formula (QTcF) \> 470 ms * NOTE: In case of clinically significant ECG abnormalities, including a QTcF value \> 470 ms, 2 additional 12-lead ECGs should be obtained over a brief period (within 30 minutes) to confirm the finding * Immunocompromised and known to be human immunodeficiency virus (HIV) positive and currently receiving antiretroviral therapy * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis (TB) testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\] result), hepatitis C. Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA) * Current or prior use of immunosuppressive medication =\< 14 days prior to registration. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., computed tomography \[CT\] scan premedication) * Receipt of live attenuated vaccine =\< 30 days prior to registration. Note: Patients, if enrolled, should not receive live vaccine whilst receiving investigational product (IP) and up to 30 days after the last dose of durvalumab and regorafenib * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy * Known allergy or hypersensitivity to any of the study drugs or any of the study drug excipients * Prior randomization or treatment in a previous durvalumab clinical study regardless of treatment arm assignment