ESK981 and Nivolumab for the Treatment of Metastatic Castration Resistant Prostate Cancer

Inclusion Criteria: * Eastern Cooperative Group (ECOG) performance status =\< 1 * Recovery to baseline or =\< grade 1 Common Terminology Criteria for Adverse Events (CTCAE) version (v.)5 from toxicities related to any prior treatments, unless adverse event (AE)(s) are clinically non-significant and/or stable on supportive therapy * Absolute neutrophil count (ANC) \>= 1.5 K/mm\^3 * Hemoglobin (Hgb) \>= 9 g/dL * Platelets (Plt) \>= 100,000/mm\^3 * Serum creatinine =\< 1.5 times the upper limit of normal OR creatinine clearance \> 30 mL/min by Cockcroft-Gault formula * Total bilirubin =\< 1.5 x upper limit of normal (ULN) * Aspartate aminotransferase (AST) =\< 2.5 x ULN (=\< 5 x ULN with known hepatic metastases) * Alanine aminotransferase (ALT) =\< 2.5 x ULN (=\< 5 x ULN with known hepatic metastases) * Prothrombin time (PT) and activated partial thromboplastin time (aPTT) levels =\< 1.5 x ULN (If patient is receiving anticoagulation that is expected to alter these levels, should be in targeted therapeutic range for that agent) * Patient must have progressive disease while receiving androgen deprivation therapy (ADT) defined by any one of the following as per the PCWG3 criteria for PSA, measurable or non-measurable (bone) disease and must have a castrate serum testosterone level (i.e. =\< 50 ng/dL) at screening: * PSA: At least two consecutive rises in serum PSA, obtained at a minimum of 1-week intervals, with the final value \>= 2.0 ng/mL * Measurable disease (by Response Evaluation Criteria in Solid Tumors \[RECIST\] 1.1): \>= 20% increase (with an absolute increase of at least 5 mm) in the sum of diameters of all measurable lesions or the development of one or more new lesions. The short axis of a target lymph node must be more than 15 mm to be assessed for change in size * Non-measurable (bone) disease: The appearance of two or more new areas of uptake on bone scan consistent with metastatic disease compared to previous imaging during castration therapy. The increased uptake of pre-existing lesions on bone scan will not be taken to constitute progression, and ambiguous results must be confirmed by other imaging modalities (e.g. X-ray, computed tomography \[CT\] or magnetic resonance imaging \[MRI\]) * Metastatic prostate cancer (M1) as documented by appropriate medical imaging (i.e. CT-Scan, positron emission tomography \[PET\] scan or bone scan) * Progression on a hormonal agent (abiraterone/enzalutamide) and on a chemotherapy agent (docetaxel and/or cabazitaxel) in the metastatic castration resistant setting as per PCWG3 criteria * Progression on chemotherapy (e.g. docetaxel, cabazitaxel) in the metastatic castration resistant setting. Progression of disease within 6 months of completing docetaxel in the metastatic castrate-sensitive setting is acceptable * Have signed an informed consent document indicating that the subject understands the purpose of and procedures required for the study and are willing to participate in the study * Be willing and able to adhere to the prohibitions and restrictions specified in this protocol * Willingness to use contraception by a method that is deemed effective by the investigator throughout the treatment period and for at least 30 days following the last dose of therapy * Willingness and ability to comply with study procedures and follow-up examination * Able to swallow and retain oral medication * Willingness and ability to undergo mandatory tumor biopsy at baseline and at the cycle 3 visit * Willingness and ability to undergo mandatory whole blood sample collections at baseline, weeks 2-4 in the first cycle, and then monthly Exclusion Criteria: * Systemic therapy (other than a gonadotrophin releasing hormone \[GnRH\] agonist/antagonist) for CRPC within the past two weeks from cycle 1/day 1 including: * CYP-17 inhibitors (e.g. ketoconazole, abiraterone) * Antiandrogens (e.g. bicalutamide, nilutamide) * Second generation antiandrogens (e.g. enzalutamide, ARN-509, galeterone) * Immunotherapy (e.g. sipuleucel-T, ipilimumab) * Chemotherapy (e.g. docetaxel, cabazitaxel) * Prior radiopharmaceutical therapy (e.g. radium-223, strontium-89, samarium-153, etc.) within the past year * Have any condition that, in the opinion of the investigator, would compromise the well-being of the subject or the study or prevent the subject from meeting or performing study requirements * The patient is currently on warfarin or heparin therapy * The patient has any pre-existing coagulopathy, recent hemoptysis, gross hematuria or gastrointestinal bleeding * The patient has a history of a clinically significant cardiovascular or cerebrovascular event within 3 months prior to study entry * The patient has uncontrolled hypertension defined as a blood pressure measurement greater than 150 mm Hg systolic or 90 mm Hg diastolic with medication * The patient has previously been enrolled in the study or received ESK981 * The patient has known hypersensitivity to gelatin or lactose monohydrate * The patient has taken a medication known to be a potent inducer of CYP1A2, CYP2C8, or CYP3A4 within 4 weeks prior to the first dose of study drug * Prior treatment with anti-PD/PD-L1/CTLA4/IDO antibody * Untreated brain metastases or spinal cord compression * Major surgical procedure or significant traumatic injury within 6 weeks prior to study registration. (\> 6 weeks prior to registration is permitted as long as they have fully recovered from any such procedure) * History of another primary malignancy except for: malignancy treated with curative intent and no known active disease for \>= 5 years, adequately treated non-melanoma skin cancer without evidence of disease, adequately treated carcinoma in situ without evidence of disease * Angina, myocardial infarction symptomatic congestive heart failure, cerebrovascular accident, transient ischemic attack, arterial embolism, pulmonary embolism, percutaneous angioplasty or Coronary arterial bypass surgery within the past 3 months * The patient has received any investigational drug within 28 days prior to registration or 5 half-lives of the investigational drug, whichever is sooner