Sirolimus and Vismodegib in Treating Patients With Solid Tumors or Pancreatic Cancer That is Metastatic or Cannot Be Removed By Surgery

Inclusion Criteria: * COHORT I (DOSE ESCALATION): histologic proof of cancer that is now unresectable, not amenable to any other standard therapies, or patient refuses standard therapy * COHORT II (MTD): metastatic adenocarcinoma of the pancreas and tumor amenable to biopsies; prior systemic treatment for metastatic disease is allowed * Absolute neutrophil count (ANC) =\> 1500/uL * Platelet \>= 100,000/uL * Total bilirubin =\< upper limit of normal (ULN) Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) =\< 3 x ULN * Creatinine =\< 1.5 x ULN * Hemoglobin \>= 9.0 g/dL * Prothrombin time (PT)/international normalized ratio (INR) \< 1.25 x ULN (Cohort II \[MTD\] only) * Cholesterol \< Common Terminology Criteria for Adverse Events (CTCAE) grade 3 * Triglycerides \< CTCAE grade 2 * Magnesium \>= lower limit of normal (LLN) and =\< ULN * Ability to provide informed consent * Willing to return to Mayo Clinic for follow up * Life expectancy \>= 12 weeks * Cohort II (MTD) only - Translational Research: Willing to provide the biologic specimens as required by the protocol; Note: this is part of the mandatory translational research component * Women of childbearing potential only: Negative serum pregnancy test done =\< 7 days prior to registration; NOTE: female subjects who are pregnant or nursing are excluded from this study; there is no specific mitigation strategy for vismodegib toxicity; however, male patients should be made aware of it during the consent process; although this effect is expected to be reversible with discontinuation of dosing, long-term effects on male fertility cannot be excluded at this time * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1 or 2 * Able to swallow or have medication administered through a G-tube and absorb the medication * Participant agrees to use acceptable form of contraception during the study and for up to 7 months after last study drug dose * Acceptable forms of contraception: * Latex condom (always used with spermicide) * Diaphragm (always used with spermicide) * Cervical cap (always used with spermicide) * Acceptable forms of secondary contraception, when used along with a barrier method: * Hormonal contraception methods, including pills, patches, rings, or injections except progestin-only containing pills (i.e. "Mini-pill") * Tubal ligation * Partner's vasectomy * Intrauterine device (non-progesterone T) * Vaginal sponge (containing spermicide) * Other acceptable forms: * 100% commitment to abstinence * Unacceptable forms of contraception for women of childbearing potential: * Oral contraception containing progestins only * Intrauterine device (IUD) progesterone T * Female condom * Natural family planning (rhythm method) or breastfeeding * Fertility awareness * Withdrawal * Cervical shield * Willing not to smoke * Willing to complete a pill diary each day Exclusion Criteria: * COHORT I (DOSE ESCALATION): Known standard therapy for the patient's disease that is potentially curative or definitely capable of extending life expectancy * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, diabetes mellitus or psychiatric illness/social situations that would limit compliance with study requirements * Any of the following prior therapies: * Chemotherapy =\< 4 weeks prior to registration * Mitomycin C/nitrosoureas =\< 6 weeks prior to registration * Immunotherapy =\< 4 weeks prior to registration * Biologic therapy =\< 4 weeks prior to registration * Radiation therapy =\< 4 weeks prior to registration * Radiation to \> 25% of bone marrow * Failure to fully recover from acute, reversible effects of prior chemotherapy regardless of interval since last treatment * New York Heart Association classification III or IV * Uncontrolled Seizure Disorder * Central nervous system (CNS) metastases if not stable for at least 2-3 months based on imaging, clinical assessment, use of steroids, or seizure disorder * Any of the following: * Pregnant women * Nursing women * This study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown * Other concurrent chemotherapy, immunotherapy, radiotherapy, any ancillary therapy considered investigational (utilized for a non-Food and Drug Administration \[FDA\]-approved indication and in the context of a research investigation) or receiving any other investigational agent which would be considered as a treatment for the primary neoplasm * Receiving any medications or substances that are strong or moderate inhibitors of cytochrome P450 3A4 (CPY450 3A4); use of the following strong or moderate inhibitors are prohibited: * Strong Inhibitors of CYP3A4: Indinavir, Nelfinavir, Ritonavir, Clarithromycin, Itraconazole, Ketoconazole, Nefazodone, Saquinavir, Telithromycin * Moderate Inhibitors of CYP3A4: Aprepitant, Erythromycin, Fluconazole, Grapefruit juice, Verapamil, Diltiazem * Receiving any medications or substances that are inducers of CYP450 3A4 * Inducers of CYP3A4: Efavirenz, Nevirapine, Carbamazepine, Modafinil, Phenobarbital, Phenytoin, Pioglitazone, Rifabutin, Rifampin, St. John's wort * Receiving any medications or substances that are strong or moderate inhibitors of CYP450 CYP2C8 * Strong or Moderate Inhibitors of CYP2C8: Gemfibrozil, Trimethoprim * Receiving any medications or substances that are inducers of CYP450 2C8 * Inducer of CYP2C8: Rifampin * Receiving any medications or substances that are strong or moderate inhibitors of CYP450 CYP2C9 * Strong or Moderate Inhibitors of CYP2C9: Fluconazole, Amiodarone * Receiving any medications or substances that are inducers of CYP450 2C9 * Inducers of CYP2C9: Rifampin, Secobarbital * Immunocompromised patients (other than that related to the use of corticosteroids) including patients receiving highly active antiretroviral therapy (HAART) treatment * Active other malignancy, excepting non-melanotic skin cancer or carcinoma-in situ (e.g. of cervix, breast prostate); if there is a history of prior malignancy, they must not be receiving other specific treatment (other than hormonal therapy) for their cancer * History of myocardial infarction =\< 6 months, or congestive heart failure requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias * Abnormalities of the cornea based on history (e.g., dry eye syndrome, Sjogren's syndrome), congenital abnormality (e.g., Fuch's dystrophy), abnormal slit-lamp examination using a vital dye (e.g., fluorescein, Bengal Rose), and/or an abnormal corneal sensitivity test (Schirmer test or similar tear production test) * Prior therapy with a hedgehog inhibitor